Optimal alphabet for single text compression

A text written using symbols from a given alphabet can be compressed the Huffman code, which minimizes length of encoded text. It is necessary, however, to employ text-specific codebook, i.e. symbol-codeword dictionary, decode original Thus, compression performance should evaluated by full code length, plus codebook. We studied several alphabets for compressing texts – letters, n-grams syllables, words, and phrases. If only sufficiently short are retained, an letters or two-grams optimal. For majority Project Gutenberg texts, best (the one that length) syllables depending on representation Letter 3 4-grams, having average comparable syllables/words, perform noticeably worse than words. Word 2-grams also never alphabet, account very large show codebook important switching naive compact significantly improves matters with number symbols, most notably meaning-expressing elements language (syllables words) provide alphabet.